Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms

While Amazon’s Mechanical Turk (AMT) helped launch online crowd work in 2005, a myriad of newer vendors now offer alternative feature sets and workflow models for accomplishing quality crowd work. Unfortunately, research on crowd work has remained largely focused on AMT, with little exploration of other platforms. Such near-exclusive focus on AMT risks its particular vagaries and limitations overly shaping research questions and directions, as well as our understanding of broader capabilities and limitations of crowd work. To address this, we present a qualitative content analysis of seven alternative platforms. After organizing prior AMT studies around a set of key problem types encountered, we define our process for inducing categories for qualitative assessment of platforms. We then contrast the key problem types with AMT vs. platform features from content analysis, informing both methodology of use and directions for future research. Our cross-platform analysis represents the only such study by researchers for researchers, intended to enrich diversity of research on crowd work and accelerate progress.